The emergent algebraic structure of RNNs and embeddings in NLP
نویسنده
چکیده
We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.
منابع مشابه
Label-Dependencies Aware Recurrent Neural Networks
In the last few years, Recurrent Neural Networks (RNNs) have proved effective on several NLP tasks. Despite such great success, their ability to model sequence labeling is still limited. This lead research toward solutions where RNNs are combined with models which already proved effective in this domain, such as CRFs. In this work we propose a solution far simpler but very effective: an evoluti...
متن کاملToward Mention Detection Robustness with Recurrent Neural Networks
One of the key challenges in natural language processing (NLP) is to yield good performance across application domains and languages. In this work, we investigate the robustness of the mention detection systems, one of the fundamental tasks in information extraction, via recurrent neural networks (RNNs). The advantage of RNNs over the traditional approaches is their capacity to capture long ran...
متن کاملMimicking Word Embeddings using Subword RNNs
Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using distributional information obtained from unlabeled data. However, the effectiveness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist. In this paper, we present MIMICK, an approach to generating OOV word em...
متن کاملAll Health Partnerships, Great and Small: Comparing Mandated With Emergent Health Partnerships; Comment on “Evaluating Global Health Partnerships: A Case Study of a Gavi HPV Vaccine Application Process in Uganda”
The plurality of healthcare providers and funders in low- and middle-income countries (LMICs) has given rise to an era in which health partnerships are becoming the norm in international development. Whether mandated or emergent, three common drivers are essential for ensuring successful health partnerships: trust; a diverse and inclusive network; and a clear governance structure. Mandated and ...
متن کاملFine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings
The tasks in fine-grained opinion mining can be regarded as either a token-level sequence labeling problem or as a semantic compositional task. We propose a general class of discriminative models based on recurrent neural networks (RNNs) and word embeddings that can be successfully applied to such tasks without any taskspecific feature engineering effort. Our experimental results on the task of...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2018